Method and device for dual-light image integration, and unmanned aerial vehicle

ABSTRACT

Embodiments of the present invention relate to a dual-light image integration method, a dual-light image integration device and an unmanned aerial vehicle (UAV). The method includes: receiving first image data from a first image device and second image data from a second image device; separately storing the first image data and the second image data; combining the first image data and the second image data to composite third image data; and transmitting the third image data. According to the present invention, dual-light images are processed by using different data processing methods depending on subsequent operations, which can maintain synchronization of the dual-light images during image transmission while avoiding information loss during image storage, thereby well satisfying user needs.

CROSS-REFERENCE

This application is a continuation application of InternationalApplication No. PCT/CN2019/111708, filed on Oct. 17, 2019, which claimspriority to Chinese Patent Application No. 201811318227.5, filed on Nov.7, 2018, which are incorporated herein by reference in its entirety.

BACKGROUND Technical Field

The present invention relates to the technical field of unmanned aerialvehicles (UAV), and in particular, to a dual-light image integrationmethod, a dual-light image integration device and a UAV.

Related Art

Dual-light images refer to multi-channel images obtained throughintegration of photographing or recording a video of the same picture bytwo or more types of image collection devices, for example, a dual-lightphotographing system composed of both a visible light imaging devicesuch as a visible light lens and an infrared thermal imaging device.

The dual-light photographing system can simultaneously obtain aninfrared imaging image and a visible light image. The two images haverespective characteristics that can complement and cooperate with eachother to provide more functional options for subsequent processing.

However, different types of image data captured by the existingdual-light photographing system still have some shortcomings in thestorage, transmission and compositing process, which restricts theperformance of subsequent image transmission and image storage. How toimprove a data processing strategy of the dual-light photographingsystem capturing the dual-light image to avoid the defects of imageinformation loss and poor synchronization of image transmission is aproblem that needs to be solved urgently.

SUMMARY

In order to solve the above technical problem, embodiments of thepresent invention provide a dual-light image integration method, adual-light image integration device and a UAV that can avoid informationloss while ensuring synchronization performance of image transmission.

In order to resolve the above technical problem, the embodiments of thepresent invention provide the following technical solutions. Adual-light image integration method is provided, including:

receiving first image data from a first image device and second imagedata from a second image device;

separately storing the first image data and the second image data;

combining the first image data and the second image data to compositethird image data; and

transmitting the third image data.

Optionally, the third image data is a picture-in-picture image includinga primary display picture and a secondary display picture superimposedon the primary display picture.

Optionally, the step of combining the first image data and the secondimage data to composite the third image data includes:

scaling the first image data to a size corresponding to a first videopicture and scaling the second image data to a size corresponding to asecond video picture;

with the first video picture as the primary display picture,superimposing the second video picture as the secondary display pictureon the primary display picture; or

with the second video picture as the primary display picture,superimposing the first video picture as the secondary display pictureon the primary display picture; and

generating the picture-in-picture image including the primary displaypicture and the secondary display picture.

Optionally, the third image data is a fused image obtained by fusing,pixel by pixel, the first image data and the second image data.

Optionally, the step of combining the first image data and the secondimage data to composite the third image data includes:

determining a first image property of the first image data at a k^(th)pixel position and a second image property of the second image data atthe k^(th) pixel position according to gray values of the first imagedata and the second image data at each pixel position;

comparing the first image property with the second image property todetermine the gray value at the k^(th) pixel position, where k is aninteger from 1 to K, where K is a number of pixels of the first imagedata and the second image data; and

obtaining the fused image according to the gray values at all pixelpositions.

Optionally, the step of determining the first image property of thefirst image data at the k^(th) pixel position and the second imageproperty of the second image data at the k^(th) pixel position accordingto the gray values of the first image data and the second image data ateach pixel position includes:

determining, through a preset detection window, whether smoothness atthe k^(th) pixel position exceeds a preset smoothness threshold, where

if so, the image properties at the k^(th) pixel position are determinedto be smooth, and

if not, the image properties at the k^(th) pixel position are determinedto be non-smooth.

Optionally, the step of comparing the first image property with thesecond image property to determine the gray value at the k^(th) pixelposition includes:

when the first image property and the second image property are bothsmooth, determining that the gray value at the k^(th) pixel position isa mean value of the gray values of the first image data and the secondimage data at the k^(th) pixel position;

when the first image property is smooth and the second image property isnon-smooth, determining that the gray value at the k^(th) pixel positionis the gray value of the second image data at the k^(th) pixel position;and

when the first image property is non-smooth and the second imageproperty is smooth, determining that the gray value at the k^(th) pixelposition is the gray value of the first image data at the k^(th) pixelposition.

Optionally, the comparing the first image property with the second imageproperty to determine the gray value at the k^(th) pixel positionincludes:

when the first image property and the second image property are bothnon-smooth, comparing the gray values of the first image data and thesecond image data at the k^(th) pixel position; and

selecting a larger one of the gray values as the gray value at thek^(th) pixel position.

Optionally, the step of determining, through the preset detectionwindow, whether the smoothness at the k^(th) pixel position exceeds thepreset smoothness threshold includes:

calculating differences between the gray value of the first image dataat the k^(th) pixel position and gray values at a plurality of adjacentpixel positions, where the adjacent pixel positions are selected byusing the detection window; and

determining whether the differences from the gray values at theplurality of adjacent pixel positions are all less than a first presetsmoothness threshold, where

if so, the first image property at the k^(th) pixel position isdetermined to be smooth, and

if not, the first image property at the k^(th) pixel position isdetermined to be non-smooth.

Optionally, the step of determining, through the preset detectionwindow, whether the smoothness at the k^(th) pixel position exceeds thepreset smoothness threshold includes:

calculating differences between the gray value of the second image dataat the k^(th) pixel position and gray values at a plurality of adjacentpixel positions, where the adjacent pixel positions are selected byusing the detection window; and

determining whether the differences from the gray values at theplurality of adjacent pixel positions are all less than a second presetsmoothness threshold, where

if so, the second image property at the k^(th) pixel position isdetermined to be smooth, and

if not, the second image property at the k^(th) pixel position isdetermined to be non-smooth.

Optionally, the method further includes: performing one or more visualprocessing operations on the fused image.

Optionally, the first image data is infrared thermal imaging image data,and the second image data is visible light image data.

Optionally, the method further includes: encoding the first image data,the second image data and the third image data by using a presetencoding algorithm.

In order to resolve the above technical problem, the embodiments of thepresent invention further provide the following technical solution. Adual-light image integration device is provided,

including: a receiving port, a compositing module, an output port and amass memory, where

the receiving port is configured to receive first image data from afirst image device and second image data from a second image device;

the compositing module is configured to perform the step of combiningthe first image data and the second image data to composite the thirdimage data in the dual-light image integration method;

the receiving port is configured to output the third image data; and

the mass memory is configured to separately store the first image dataand the second image data.

Optionally, the device further includes an encoding module

configured to encode the first image data, the second image data and thethird image data by using a preset encoding algorithm, where

the mass memory is further configured to separately store the encodedfirst image data and the encoded second image data.

Optionally, the encoding module is specifically configured to: when thefirst image data and the second image data are pictures, encode thefirst image data, the second image data and the third image data throughJPEG encoding, DNG encoding or TIFF encoding; and when the first imagedata and the second image data are videos, encode the first image data,the second image data and the third image data through H265 encoding orH264 encoding. The mass memory is an SD card.

To resolve the foregoing technical problem, an embodiment of the presentinvention further provides the following technical solution: a UAV. TheUAV includes:

a UAV body carrying a first image device and a second image devicearranged side by side;

the dual-light image integration device, where the dual-light imageintegration device is connected to the first image device and the secondimage device and is configured to receive the first image data and thesecond image data; and

an image transmission apparatus connected to the dual-light imageintegration device and configured to transmit the third image datacomposited by the dual-light image integration device to a groundterminal.

Optionally, the first image device is an infrared camera, and the secondimage device is a high-definition camera.

Compared with the prior art, according to the dual-light imageintegration method provided in the embodiments of the present invention,dual-light images (that is, the first image data and the second imagedata) are processed by using different data processing methods dependingon subsequent applications, which can maintain synchronization of thedual-light images during image transmission while avoiding informationloss during image storage, thereby well satisfying user needs.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments are exemplarily described with reference to thecorresponding figures in the accompanying drawings, and the descriptionsare not to be construed as limiting the embodiments. Elements in theaccompanying drawings that have same reference numerals are representedas similar elements, and unless otherwise particularly stated, thefigures in the accompanying drawings are not drawn to scale.

FIG. 1 is a schematic diagram of an application environment according toan embodiment of the present invention.

FIG. 2 is a structural block diagram of an electronic computing platformaccording to an embodiment of the present invention.

FIG. 3 is a schematic diagram of application of a dual-light imageaccording to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a dual-light image integration deviceaccording to an embodiment of the present invention.

FIG. 5 is a flowchart of a dual-light image integration method accordingto an embodiment of the present invention.

FIG. 6 is a schematic diagram of a picture-in-picture image according toan embodiment of the present invention.

FIG. 7 is a schematic diagram of a picture-in-picture image according toanother embodiment of the present invention.

FIG. 8 is a flowchart of a method for generating a fused image accordingto an embodiment of the present invention.

FIG. 9 is a flowchart of an image fusing method according to anembodiment of the present invention.

FIG. 10 is a schematic diagram of a detection window according to anembodiment of the present invention.

FIG. 11 is a schematic diagram of the detection window for imageintegration according to an embodiment of the present invention.

DETAILED DESCRIPTION

For ease of understanding the present invention, the present inventionis described in more detail below with reference to the accompanyingdrawings and specific embodiments. It should be noted that, when acomponent is expressed as “being fixed to” another component, thecomponent may be directly on the another component, or one or moreintermediate components may exist between the component and the anothercomponent. When one component is expressed as “being connected to”another component, the component may be directly connected to theanother component, or one or more intermediate components may existbetween the component and the another component. In the description ofthis specification, orientation or position relationships indicated bythe terms such as “up”, “down”, “inside”, “outside” and “bottom” arebased on orientation or position relationships shown in the accompanyingdrawings, and are used only for ease and brevity of illustration anddescription of the present invention, rather than indicating or implyingthat the mentioned apparatus or component must have a particularorientation or must be constructed and operated in a particularorientation. Therefore, such terms should not be construed as limitingof the present invention. In addition, terms “first”, “second” and“third” are only used to describe the objective and cannot be understoodas indicating or implying relative importance.

Unless otherwise defined, meanings of all technical and scientific termsused in the present invention are the same as that usually understood bya person skilled in the technical field to which the present inventionbelongs. In the present invention, terms used in the specification ofthe present invention are merely intended to describe objectives of thespecific embodiments, but are not intended to limit the presentinvention. A term “and/or” used in this specification includes any orall combinations of one or more related listed items.

In addition, technical features involved in different embodiments of thepresent invention described below may be combined together if there isno conflict.

Dual-light images generally refer to images obtained through collectionby a dual-light image collection device such as a dual-light camera thatincludes two or more types of lenses. For ease of description, thedual-light camera in the embodiment of the present invention is an imagecollection device including an infrared camera and a visible lightcamera, which obtains an infrared thermal imaging image and a visiblelight high-definition image for the same captured picture throughcollection.

FIG. 1 shows an application environment according to an embodiment ofthe present invention. As shown in FIG. 1 , the application environmentincludes a UAV 10, a smart terminal 20 and a wireless network 30.

The UAV 10 may be any type of power-driven UAV, including, but notlimited to, a quadrotor UAV, a fixed-wing aircraft and a helicoptermodel. In this embodiment, the quadrotor UAV is used as an example fordescription.

The UAV 10 may have a corresponding volume or power according to anactual requirement, to provide a load capacity, a flight speed and aflight mileage that can meet a use requirement. One or more functionalmodules may further be added to the UAV to enable the UAV to implementcorresponding functions.

For example, in this embodiment, the UAV 10 may be equipped with a fixedbracket via a gimbal or the like, and carries a dual-light camera forcollecting dual-light images. Correspondingly, the UAV 10 may also beequipped with an image transmission apparatus to upload the dual-lightimage collected by the dual-light camera to a device connected to theUAV 10.

The UAV 10 includes at least one main control chip, which serves as acontrol unit of the UAV 10 for flight and data transmission andintegrates one or more modules to execute corresponding logic controlprograms.

For example, in some embodiments, the main control chip may include thedual-light image integration device for performing data processing onthe dual-light image and subsequent application, so as to implement theprocessing, transmission or storage of the dual-light image.

FIG. 2 is a structural block diagram of an electronic computing platformaccording to an embodiment of the present invention. The electroniccomputing platform may be configured to implement functions of all orpart of functional modules in the main control chip. As shown in FIG. 2, the electronic computing platform 100 may include: a processor 110, amemory 120 and a communication module 130.

Any two of the processor 110, the memory 120 and the communicationmodule 130 are communicatively connected by a bus.

The processor 110 may be any type of processor that has one or moreprocessing cores, which can perform single-threaded or multi-threadedoperations and is configured to analyze instructions to performoperations such as obtaining data, performing logical operationfunctions, and delivering operation processing results.

The memory 120 is used as a non-volatile computer-readable storagemedium, for example, at least one magnetic disk storage device, a flashmemory, a distributed storage device remotely disposed relative to theprocessor 110 or another non-volatile solid-state storage device.

The memory 120 may have a program storage region used to store anon-volatile software program, a non-volatile computer-executableprogram and a module to be invoked by the processor 110 to enable theprocessor 110 to perform one or more method steps. The memory 120 mayfurther have a data storage region used to store the operationprocessing result delivered and outputted by the processor 110.

The communication module 130 is a functional module configured toestablish a communication connection and provide a physical channel. Thecommunication module 130 may be any type of wireless or wiredcommunication module, including but not limited to a Wi-Fi module or aBluetooth module.

The smart terminal 20 may be any type of smart device configured toestablish a communication connection to the UAV, for example, a mobilephone, a tablet computer, a smart remote control or the like. The smartterminal 20 may be equipped with one or more types of different userinteraction apparatuses configured to acquire a user instruction orpresent and feed back information to the users.

The interaction apparatuses include, but not limited to, a button, adisplay screen, a touchscreen, a speaker and a remote control joystick.For example, the smart terminal 20 may be equipped with a touch displayscreen. Through the touch display screen, a remote control instructionfor the UAV is received from a user, and image information obtainedthrough aerial photography by the dual-light camera is presented to theuser. The user may further switch the image information currentlydisplayed on the display screen through a remote touch screen.

In some embodiments, the existing image visual processing technology mayfurther be fused between the UAV 10 and the smart terminal 20 to furtherprovide more intelligent services. For example, the UAV 10 may collectimages through a dual-light camera, and the smart terminal 20 analyzesthe images, so as to implement gesture control for the UAV 10 by theuser.

The wireless network 30 may be a wireless communication networkconfigured to establish a data transmission channel between two nodesbased on any type of data transmission principle, for example, aBluetooth network, a Wi-Fi network, a wireless cellular network, or acombination thereof located in different signal frequency bands.

The application environment shown in FIG. 1 only shows the applicationof the dual-light cameras on the UAV. Those skilled in the art canunderstand that the dual-light camera may further be carried on othertypes of mobile vehicles (such as a remote control car) to perform thesame functions. The inventive idea about the dual-light camera and thedual-light image disclosed in the embodiments of the present inventionis not limited to the application on the UAV shown in FIG. 1 .

FIG. 3 shows an application process of a dual-light image according toan embodiment of the present invention. In the embodiment shown in FIG.3 , the dual-light image includes first image data A and second imagedata B. The first image data A and the second image data B may becombined and outputted as single-channel third image data.

As shown in FIG. 3 , the first image data A and the second image data Bare usually collected by the dual-light camera by using the followingflow directions: transmitting the first image data and the second imagedata to a mass storage for storage (210), transmitting the first imagedata and the second image data to a display device for display to theuser (220) or performing image visual processing (230).

FIG. 4 is an integration device for performing the dual-light imageapplication process shown in FIG. 3 according to an embodiment of thepresent invention, and may be equipped with corresponding hardware andsoftware units to implement the data processing process shown in FIG. 2. The integration device may specifically be implemented by any existingtype of processor chip, which may serve as a separate chip or one of thefunctional modules to be integrated into a flight control chip of theUAV.

As shown in FIG. 4 , the integration device may include: a receivingport 41, a compositing module 42, an output port 43 and a mass memory44.

The receiving port 41 is an input port connected to a dual-light cameraor a data source, which is configured to receive first image data from afirst image device and second image data from a second image device.

The compositing module 42 is a functional module for performing imagedata composition, which may be implemented by software, hardware, or acombination of software and hardware, and is configured to combine thefirst image data and the second image data to obtain single-channelimage data for output.

In some embodiments, the third image data may be a picture-in-pictureimage including a primary display picture and a secondary displaypicture superimposed on the primary display picture.

Correspondingly, the compositing module 42 may be specificallyconfigured to: first scale the first image data to a size correspondingto a first video picture and scale the second image data to a sizecorresponding to a second video picture; with the first video picture asthe primary display picture, superimpose the second video picture as thesecondary display picture on the primary display picture; or with thesecond video picture as the primary display picture, superimpose thefirst video picture as the secondary display picture on the primarydisplay picture; and finally generate the picture-in-picture imageincluding the primary display picture and the secondary display picture.

In some other embodiments, the third image data is a fused imageobtained by fusing, pixel by pixel, the first image data and the secondimage data.

Correspondingly, the compositing module 42 may be specificallyconfigured to: first determine a first image property of the first imagedata at a k^(th) pixel position and a second image property of thesecond image data at the k^(th) pixel position according to gray valuesof the first image data and the second image data at each pixelposition; then compare the first image property with the second imageproperty to determine the gray value at the k^(th) pixel position, wherek is an integer from 1 to K, where K is a number of pixels of the firstimage data and the second image data; and finally obtain the fused imageaccording to the gray values at all pixel positions.

Those skilled in the art may choose to use corresponding software,hardware, or a combination of software and hardware to implement thefunctions (for example, one or more comparison circuits, an image windowprocessing circuit or the like) according to the functional steps to beperformed by the compositing module 42 disclosed in the embodiment ofthe present utility model. According to the functional steps to beimplemented, the method of selecting and designing a hardware circuit iswell-known to those skilled in the art and is common knowledge in thetechnical field, and the details are not described herein.

The receiving port 43 is a data output port, which may be connected to aplurality of subsequent processing modules to output the third imagedata obtained by combining the first image data and the second imagedata.

The mass memory 44 may be any suitable type of non-volatile storagedevice, which is configured to provide enough storage space to save alloriginal data obtained through collection, including but not limited toan SD card, an SSD hard disk, a mechanical hard disk, a flash memory orthe like. In this embodiment, the mass memory 44 adopts an SD card.

A suitable number of the mass memory 44 may be selected according toactual needs. For example, the number may be set to two, and the twomass memories are respectively configured to store the first image dataor the second image data. Certainly, the mass memory 44 may alsodirectly use storage spaces or storage devices provided by other systemsin the UAV. For example, the mass memory may be directly stored in astorage device of an image transmission apparatus of the UAV.

In addition to the three basic flow directions of image data, some imageprocessing steps may further be added or reduced to meet differentneeds.

For example, the image processing step may further include an encodingoperation on image data to reduce an amount of data required fortransmission and storage. That is, encode, by using a preset encodingalgorithm, the first image data, the second image data and the thirdimage data obtained through combination (240).

The encoded first image data and second image data may be stored in amass storage. The encoded third image data may be provided for visualprocessing or to a smart terminal for display to a user.

The preset encoding algorithm may specifically adopt any type ofencoding algorithm, or one or more encoding algorithms may be selectedfor encoding according to a difference in the image data. For example,when the first image data and the second image data are pictures, apreset encoding algorithm may be JPEG encoding, DNG encoding or TIFFencoding. However, when the first image data and the second image dataare videos, the preset encoding algorithm is H265 encoding or H264encoding.

Correspondingly, the integration device may further include an encodingmodule 45. The encoding module is disposed between the output port andthe mass memory and is configured to perform the above encodingoperation.

In a traditional processing method, the first image data A and thesecond image data B are usually combined into single-channel third imagedata to be provided to a subsequent processing module (such as a massmemory or the like). However, upon combination, the image informationstored in the mass storage device overlaps, and the original imageinformation cannot be separated, resulting in deficiencies in the storedimage information.

However, in some other dual-light camera designs, two different types oflenses and subsequent image processing structures are separatelydisposed completely. Therefore, the first image data A and the secondimage data B are transmitted independently of each other, andinconsistency is likely to occur between the first image data and thesecond image data. When a display device presents the first image data Aand the second image data B to the user, the synchronization is poor,which seriously affects user experience.

Through the dual-light image integration method provided in theembodiment of the present invention, the problems existing in the abovetraditional processing methods may be solved properly, and the problemof loss of original information and non-synchronization of the imagedisplay during storage is also solved. In this embodiment, the firstimage data is used as infrared thermal imaging image data, and thesecond image data is used as visible light data for description.

FIG. 5 shows a dual-light image integration method according to anembodiment of the present invention. As shown in FIG. 5 , the dual-lightimage integration method includes the following steps.

501: Receive first image data from a first image device and second imagedata from a second image device.

The first image data and the second image data are image data ofdifferent photographing types, for example, infrared images and visiblelight images. The first image data and the second image data correspondto each other. That is, the first image data and the second image dataare collected by two lenses arranged side by side and are recorded dataof the same picture or scene under different types of light rays.

502: Separately store the first image data and the second image data.

Upon completion of the data collection, the storage and recording of thefirst image data A and the second image data B are separately performed(flow direction 210). That is, the first image data A and the secondimage data B are respectively stored in different storage blocks ordifferent storage devices of the same storage device.

503: Combine the first image data and the second image data to compositethird image data.

The process of combining and compositing the first image data and thesecond image data refers to integrating the first image data and thesecond image data according to a certain algorithm rule. The third imagedata finally obtained through compositing is a piece of single-channelimage data, which includes information about the first image data andthe second image data.

504: Provide the third image data.

Single-channel third image data may be further provided for subsequentprocessing operations, for example, provided to an image transmissionsystem or a smart terminal for display (flow direction 220) or forvisual computing processing.

The third image data composited through the dual-light image integrationmethod provided in the embodiment of the present invention does not havethe problem of the synchronization of the dual-channel image data duringthe transmission, and has a better display effect when displayed on thedisplay device, thereby improving user experience.

Certainly, the way of compositing the third image data may be adjustedaccordingly according to different flow directions or different thirdimage data. For example, in some embodiments, the third image data maybe a picture-in-picture image displayed in the display device.

The picture-in-picture image is a special way of presenting content,which means that when the primary display picture is displayed in fullscreen, another secondary display picture that is played simultaneouslyis inserted into a part of the area of the primary display picture, sothat the user can simultaneously observe two or more video images.

Specifically, the method of obtaining the picture-in-picture imagethrough compositing in step 503 may include the following steps:

first scaling the first image data to a size corresponding to a firstvideo picture and scaling the second image data to a size correspondingto a second video picture, where

the sizes of the first video picture and the second video picture aredetermined according to actual conditions and are related to factorssuch as a screen size of the display device, a size of thepicture-in-picture, whether the video picture is selected as the primarydisplay picture and the like;

then determining which one of the first video picture and the secondvideo picture is the primary display picture according to a selectioninstruction of a user; and finally superimposing the secondary displaypicture on the primary display picture, and generating thepicture-in-picture image.

For example, as shown in FIG. 6 , with the first video picture as theprimary display picture, the second video picture as the secondarydisplay picture is superimposed on the primary display picture.

Alternatively, as shown in FIG. 7 , with the second video picture as theprimary display picture, the first video picture as the secondarydisplay picture is superimposed on the primary display picture, and thepicture-in-picture image is generated.

The primary display picture refers to a part displayed in full screen inthe picture-in-picture image, while the secondary display picture refersto a part of the display area superimposed on the part displayed in fullscreen. The user may select the first video picture or the second videopicture as the primary display picture of the picture-in-picture imageaccording to own needs. Certainly, the sizes of the primary displaypicture and the secondary display picture may further be adjustedaccordingly.

Finally, the picture-in-picture image including the primary displaypicture and the secondary display picture is to be generated. Thegenerated picture-in-picture image is a single-channel image, which isobtained upon combination of the first image data and the second imagedata. Each frame of the picture-in-picture image is complete, and theproblem of synchronization may not occur after the image is transmittedto the display device.

In some other embodiments, when flow direction 230 is provided, thethird image data may be a fused image obtained by fusing, pixel bypixel, the first image data and the second image data.

The fused image is basic image data provided for image visualprocessing, which is integrated by the first image data and the secondimage data through pixel-by-pixel fusion. The image visual processingmay be any suitable one or more image processing algorithms, includingedge detection, face recognition, smoothing processing and the like.

For the traditional fused image, after the first image data is directlycompared with the second image data pixel by pixel, a larger one istaken from two gray values as the gray value of the fused image at thepixel. Such a simple fusion method may easily lead to loss of imagedetails and affect the subsequent image visual processing process. Forexample, when a part of an area of the first image data has a richertexture and a part corresponding to the second image data is a highgray-scale area, texture of the fused image in the area is to disappearand cannot be saved.

In order to save the details of the image as much as possible andimprove accuracy of subsequent image visual processing, in someembodiments, the method steps shown in FIG. 8 may be used to obtain thefused image. As shown in FIG. 8 , the process of fusing the imagesincludes the following steps.

801: Determine a first image property of the first image data at ak^(th) pixel position and a second image property of the second imagedata at the k^(th) pixel position according to gray values of the firstimage data and the second image data at each pixel position.

The pixel positions refer to positions of the pixels in the image andare used to identify different pixels. For example, in a two-dimensionalimage, the pixel position of a pixel may be represented by an array suchas (x, y), where x is a position of the pixel in the length of theimage, and y is a position of the pixel in the width of the image.

In this embodiment, k is used to represent different pixel positions.The image properties refer to characteristics of the image at the pixelposition, including whether the texture is rich, whether the transitionis smooth, whether the position is at an edge or the like. Generally,the image properties may be calculated or defined by using a domain ofpixel positions.

802: Compare the first image property with the second image property todetermine the gray value at the k^(th) pixel position, where

k is an integer from 1 to K, where K is a number of pixels of the firstimage data and the second image data.

For a two-dimensional image, numbers of pixels included in the lengthand width are multiplied to obtain a number of pixels in thetwo-dimensional image. As described above, pixels between the firstimage data and the second image data correspond to each other and havethe same number.

803: Obtain the fused image according to the gray values at all pixelpositions.

After each pixel position is determined repeatedly and the gray valuesat the pixel positions are determined, the final fused image may beobtained and provided for the subsequent visual processing.

FIG. 9 shows a specific method for image fusion according to anembodiment of the present invention. As shown in FIG. 9 , the imagefusion method may include the following steps.

901: Set a size of a detection window. The size of detection windowrefers to a size of a window for detection and sampling duringcalculation of an image property, which is configured to define aneighborhood of a suitable size.

902: Determine, through a preset detection window, whether smoothness atthe k^(th) pixel position is less than a preset smoothness threshold. Ifso, step 903 is performed, and if not, step 904 is performed.

Specifically, an appropriate size may be set for the detection windowaccording to actual conditions. For example, as shown in FIG. 10 , adetection window with a length and a width of both 3 pixels may beprovided. Each image property is calculated according to the gray valuerelationship between the pixels in the detection window.

FIG. 10 is a schematic diagram of a detection window according to anembodiment of the present invention. As shown in FIG. 10 , in atwo-dimensional image, the size of the detection window is 3×3, thek^(th) pixel position is represented by X_(m, n), and m and nrespectively represent pixel points in a row and a column of thetwo-dimensional image.

Differences between the gray value at the k^(th) pixel position and grayvalues at a plurality of adjacent pixel positions in the detectionwindow are first sequentially calculated. That is, gray valuedifferences between X_(m−1, n) and X_(m, n), X_(m+1, n) and X_(m, n),X_(m, n-1) and X_(m, n) and X_(m, n-1) and X_(m, n), are respectivelycalculated.

Then, it is determined whether the difference from the gray valuesobtained through the above calculation is less than the presetsmoothness threshold. When the differences of all gray values are allless than the smoothness threshold, step 903 is performed, or otherwise,step 904 is performed. The smoothness threshold is an empirical value,which may be selected by a person skilled in the art according to actualimage data, data types and/or other related factors.

903: Determine that image properties at the k^(th) pixel position aresmooth.

Smoothness means that an image area at a current pixel position has animage feature with less texture, which may usually indicate that thereis less detail at the pixel.

904: Determine that the image properties at the k^(th) pixel positionare non-smooth.

Non-smoothness means that the image area at the position has a richertexture as for the defined standard. Such an image feature may indicatethat the pixel position has more details and needs to be preserved.

Step 901 to step 904 are respectively performed on the first image dataand the second image data, so as to determine whether the imageproperties at the k^(th) pixel position are smooth.

Specifically, since the types of the image data are different, differentdetermination standards or smoothness thresholds may further be usedwhen it is determined whether the first image data and the second imagedata are smooth. For example, the first smoothness threshold is used asthe first image data, and the second smoothness threshold is used as thesecond image data. A first image property of the first image data at thepixel position k (905 a) and a second image property of the second imagedata at the same pixel position (905 b) are finally obtained.

Upon determining of the first image property and the second imageproperty, the following steps may continue to be performed.

906: Determine whether the first image property and the second imageproperty are the same. If so, step 907 is performed, and if not, step908 is performed.

907: Determine whether the first image property and the second imageproperty are both smooth. If so, step 909 is performed. If not, step 910is performed.

908: Determine which one of the first image property and the secondimage property is non-smooth. If the first image property is non-smooth,step 912 is performed. If the second image property is non-smooth, step913 is performed.

909: Calculate a mean value of the gray values of the first image dataand the second image data at the k^(th) pixel position as the gray valueat the k^(th) pixel position.

910: Compare the gray values of the first image data and the secondimage data at the k^(th) pixel position.

911: Select a larger one of the two gray values as the gray value at thek^(th) pixel position.

912: Use the gray value of the second image data at the k^(th) pixelposition as the gray value at the k^(th) pixel position.

913: Use the gray value of the first image data at the k^(th) pixelposition as the gray value at the k^(th) pixel position.

The implementation of the steps shown in FIG. 9 in actual operations isdescribed in detail below with reference to specific examples. It isassumed that the first image data is visible light, the second imagedata is infrared thermal imaging image data, and the size of thedetection window is 3×3.

x_(m, n) represents gray scale of the visible light at a pixel (m, n);T₁ is the first smoothness threshold; and y_(m, n) represents gray scaleof the infrared thermal imaging image data at the pixel (m, n), T₂ isthe second smoothness threshold, and z_(m, n) is gray scale of the fusedimage at the pixel (m, n).

As shown in FIG. 11 , gray scale data including the pixel (m, n) and atotal of 9 pixels around the pixel in the first image data and thesecond image data is first collected.

Then, gray scale z_(m, n) of the fused image at the pixel (m, n) iscalculated according to the following piecewise function composed offour-segment curves.

1)

${When}\left\{ {\begin{matrix}{{❘{x_{m,n} - x_{{m - 1},n}}❘} < T_{1}} \\{{❘{x_{m,n} - x_{{m + 1},n}}❘} < T_{1}} \\{{❘{x_{m,n} - x_{m,{n - 1}}}❘} < T_{1}} \\{{❘{x_{m,n} - x_{m,{n + 1}}}❘} < T_{1}}\end{matrix}{and}\left\{ {{\begin{matrix}{{❘{y_{m,n} - y_{{m - 1},n}}❘} < T_{2}} \\{{❘{y_{m,n} - y_{{m + 1},n}}❘} < T_{2}} \\{{❘{y_{m,n} - y_{m,{n - 1}}}❘} < T_{2}} \\{{❘{y_{m,n} - y_{m,{n + 1}}}❘} < T_{2}}\end{matrix}{are}{satisfied}},{{{let}{}z_{m,n}} = {\frac{x_{m,n} + y_{m,n}}{2}.}}} \right.} \right.$

The first curve is the situation in step 909 in FIG. 9 , and thecorresponding determination condition is that both the first imageproperty and the second image property are smooth.

2)

${When}\left\{ {\begin{matrix}{{❘{x_{m,n} - x_{{m - 1},n}}❘} \geqq T_{1}} \\{{{or}{}{❘{x_{m,n} - x_{{m + 1},n}}❘}} \geqq T_{1}} \\{{{or}{❘{x_{m,n} - x_{m,{n - 1}}}❘}} \geqq T_{1}} \\{{{or}{❘{x_{m,n} - x_{m,{n + 1}}}❘}} \geqq T_{1}}\end{matrix}{and}\left\{ {{\begin{matrix}{{❘{y_{m,n} - y_{{m - 1},n}}❘} < T_{2}} \\{{❘{y_{m,n} - y_{{m + 1},n}}❘} < T_{2}} \\{{❘{y_{m,n} - y_{m,{n - 1}}}❘} < T_{2}} \\{{❘{y_{m,n} - y_{m,{n + 1}}}❘} < T_{2}}\end{matrix}{are}{satisfied}},{{{let}{}z_{m,n}} = {x_{m,n}.}}} \right.} \right.$

The second curve is the situation in step 913 in FIG. 9 , and thecorresponding determination condition is that the first image propertyis smooth and the second image property is non-smooth.

3)

${When}\left\{ {\begin{matrix}{{❘{x_{m,n} - x_{{m - 1},n}}❘} < T_{1}} \\{{❘{x_{m,n} - x_{{m + 1},n}}❘} < T_{1}} \\{{❘{x_{m,n} - x_{m,{n - 1}}}❘} < T_{1}} \\{{❘{x_{m,n} - x_{m,{n + 1}}}❘} < T_{1}}\end{matrix}{and}\left\{ {{\begin{matrix}{{❘{y_{m,n} - y_{{m - 1},n}}❘} \geqq T_{2}} \\{{{or}{❘{y_{m,n} - y_{{m + 1},n}}❘}} \geqq T_{2}} \\{{{or}{❘{y_{m,n} - y_{m,{n - 1}}}❘}} \geqq T_{2}} \\{{{or}{❘{y_{m,n} - y_{m,{n + 1}}}❘}} \geqq T_{2}}\end{matrix}{are}{satisfied}},{{{let}{}z_{m,n}} = {y_{m,n}.}}} \right.} \right.$

The third curve is the situation in step 912 in FIG. 9 , and thecorresponding determination condition is that the first image propertyis smooth and the second image property is non-smooth.

4) When the above three conditions are not satisfied, letz_(m,n)=max(x_(m,n), y_(m,n)).

The fourth curve is the situation in step 911 in FIG. 9 , and thecorresponding determination condition is that both the first imageproperty and the second image property are non-smooth.

Through the above methods, the gray value of the final pixel can becalculated according to the specific situation of each pixel position.In this way, image details included in the two types of image data canbe better reflected and retained. An information source of the finallyobtained fused image is relatively rich, and the output effect of thesubsequent vision processing algorithm is consistent, which alsofacilitates improvement of the processing performance of the visionprocessing algorithm.

According to the dual-light image integration method provided in theembodiment of the present invention, a fused image or apicture-in-picture image can be provided. The user can inputcorresponding control instructions on the smart terminal 20 to adjustthe pictures displayed on the display screen. For example, the user mayinput a switching instruction to use the infrared image as the primarydisplay picture and the visible light image as the secondary displaypicture, or may further control the smart terminal to directly displaythe fused image.

In this way, the images transmitted to the image transmission apparatusare superimposed to form single-channel image data in apicture-in-picture manner, and upon decoding and display, can bettersupport real-time viewing of the smart terminal, so as to prevent thetwo-channel images being out of sync.

A person of ordinary skill in the art may further be aware that, incombination with examples of each step of the dual-light imageintegration method described in the embodiments disclosed in thisspecification, the present application may be implemented by usingelectronic hardware, computer software, or a combination thereof. Toclearly describe interchangeability between the hardware and thesoftware, compositions and steps of each example have been generallydescribed according to functions in the foregoing descriptions. Whetherthe functions are executed in a mode of hardware or software depends onparticular applications and design constraint conditions of thetechnical solutions.

A person skilled in the art may use different methods to implement thedescribed functions for each particular application, but thisimplementation shall not be considered as going beyond the scope of thepresent invention. The computer software may be stored in acomputer-readable storage medium. When being executed, the program mayinclude the processes of the embodiments of the foregoing methods. Thestorage medium may be a magnetic disk, an optical disc, a read-onlymemory (ROM), or a random access memory (RAM).

Finally, it should be noted that the foregoing embodiments are merelyused for describing the technical solutions of the present invention,but are not intended to limit the present invention. Under the conceptof the present invention, the technical features in the foregoingembodiments or different embodiments may be combined, the steps may beimplemented in any sequence, and there may be many other changes indifferent aspects of the present invention as described above. Forbrevity, those are not provided in detail. Although the presentinvention is described in detail with reference to the foregoingembodiments, a person of ordinary skill in the art should understandthat they may still make modifications to the technical solutionsdescribed in the foregoing embodiments or make equivalent replacementsto some technical features thereof, without departing from the scope ofthe technical solutions of the embodiments of the present invention.

The invention claimed is:
 1. A dual-light image integration method,comprising: receiving first image data from a first image device andsecond image data from a second image device; separately storing thefirst image data and the second image data; combining the first imagedata and the second image data to composite third image data; andproviding the third image data; wherein the third image data is a fusedimage obtained by fusing, pixel by pixel, the first image data and thesecond image data; wherein the step of combining the first image dataand the second image data to composite the third image data comprises:determining a first image property of the first image data at a k^(th)pixel position and a second image property of the second image data atthe k^(th) pixel position according to gray values of the first imagedata and the second image data at each pixel position; comparing thefirst image property with the second image property to determine thegray value at the k^(th) pixel position, wherein k is an integer from 1to K, wherein K is a number of pixels of the first image data and thesecond image data; and obtaining the fused image according to the grayvalues at all pixel positions.
 2. The dual-light image integrationmethod according to claim 1, wherein the third image data is apicture-in-picture image comprising a primary display picture and asecondary display picture superimposed on the primary display picture.3. The dual-light image integration method according to claim 2, whereinthe step of combining the first image data and the second image data tocomposite the third image data comprises: scaling the first image datato a size corresponding to a first video picture and scaling the secondimage data to a size corresponding to a second video picture; with thefirst video picture as the primary display picture, superimposing thesecond video picture as the secondary display picture on the primarydisplay picture; or with the second video picture as the primary displaypicture, superimposing the first video picture as the secondary displaypicture on the primary display picture; and generating thepicture-in-picture image comprising the primary display picture and thesecondary display picture.
 4. The dual-light image integration methodaccording to claim 1, wherein the step of determining the first imageproperty of the first image data at the k^(th) pixel position and thesecond image property of the second image data at the k^(th) pixelposition according to the gray values of the first image data and thesecond image data at each pixel position comprises: determining, througha preset detection window, whether smoothness at the k^(th) pixelposition exceeds a preset smoothness threshold, wherein if so, the imageproperties at the k^(th) pixel position are determined to be smooth, andif not, the image properties at the k^(th) pixel position are determinedto be non-smooth.
 5. The dual-light image integration method accordingto claim 4, wherein the step of comparing the first image property withthe second image property to determine the gray value at the k^(th)pixel position comprises: when the first image property and the secondimage property are both smooth, determining that the gray value at thek^(th) pixel position is a mean value of the gray values of the firstimage data and the second image data at the k^(th) pixel position; whenthe first image property is smooth and the second image property isnon-smooth, determining that the gray value at the k^(th) pixel positionis the gray value of the second image data at the k^(th) pixel position;and when the first image property is non-smooth and the second imageproperty is smooth, determining that the gray value at the k^(th) pixelposition is the gray value of the first image data at the k^(th) pixelposition.
 6. The dual-light image integration method according to claim4, wherein the step of comparing the first image property with thesecond image property to determine the gray value at the k^(th) pixelposition comprises: when the first image property and the second imageproperty are both non-smooth, comparing the gray values of the firstimage data and the second image data at the k^(th) pixel position; andselecting a larger one of the gray values as the gray value at thek^(th) pixel position.
 7. The dual-light image integration methodaccording to claim 4, wherein the step of determining, through thepreset detection window, whether the smoothness at the k^(th) pixelposition exceeds the preset smoothness threshold comprises: calculatingdifferences between the gray value of the first image data at the k^(th)pixel position and gray values at a plurality of adjacent pixelpositions, wherein the adjacent pixel positions are selected by usingthe detection window; and determining whether the differences from thegray values at the plurality of adjacent pixel positions are all lessthan a first preset smoothness threshold, wherein if so, the first imageproperty at the k^(th) pixel position is determined to be smooth, and ifnot, the first image property at the k^(th) pixel position is determinedto be non-smooth.
 8. The dual-light image integration method accordingto claim 4, wherein the step of determining, through the presetdetection window, whether the smoothness at the k^(th) pixel positionexceeds the preset smoothness threshold comprises: calculatingdifferences between the gray value of the second image data at thek^(th) pixel position and gray values at a plurality of adjacent pixelpositions, wherein the adjacent pixel positions are selected by usingthe detection window; and determining whether the differences from thegray values at the plurality of adjacent pixel positions are all lessthan a second preset smoothness threshold, wherein if so, the secondimage property at the k^(th) pixel position is determined to be smooth,and if not, the second image property at the k^(th) pixel position isdetermined to be non-smooth.
 9. The dual-light image integration methodaccording to claim 1, further comprising: performing one or more visualprocessing operations on the fused image.
 10. The dual-light imageintegration method according to claim 1, wherein the first image data isinfrared thermal imaging image data, and the second image data isvisible light image data.
 11. The dual-light image integration methodaccording to claim 1, further comprising: encoding the first image data,the second image data and the third image data by using a presetencoding algorithm.
 12. A dual-light image integration device,comprising: a memory storing computer executable instructions; and aprocessor configured to execute the instructions to: receive first imagedata from a first image device and second image data from a second imagedevice; separately store the first image data and the second image data;combine the first image data and the second image data to compositethird image data; and provide the third image data; wherein the thirdimage data is a fused image obtained by fusing, pixel by pixel, thefirst image data and the second image data; wherein the step ofcombining the first image data and the second image data to compositethe third image data comprises: determining a first image property ofthe first image data at a k^(th) pixel position and a second imageproperty of the second image data at the k^(th) pixel position accordingto gray values of the first image data and the second image data at eachpixel position; comparing the first image property with the second imageproperty to determine the gray value at the k^(th) pixel position,wherein k is an integer from 1 to K, wherein K is a number of pixels ofthe first image data and the second image data; and obtaining the fusedimage according to the gray values at all pixel positions.
 13. Thedual-light image integration device according to claim 12, wherein theprocessor is further configured to: encode the first image data, thesecond image data and the third image data by using a preset encodingalgorithm; and separately store the encoded first image data and theencoded second image data.
 14. The dual-light image integration deviceaccording to claim 12, wherein when the first image data and the secondimage data are pictures, the first image data, the second image data andthe third image data are encoded through JPEG encoding, DNG encoding orTIFF encoding; and when the first image data and the second image dataare videos, the first image data, the second image data and the thirdimage data are encoded through H265 encoding or H264 encoding.
 15. Anunmanned aerial vehicle (UAV), comprising: a UAV body carrying a firstimage device and a second image device arranged side by side; adual-light image integration device, wherein the dual-light imageintegration device is connected to the first image device and the secondimage device, and the dual-light image integration device comprises: amemory storing computer executable instructions; and a processorconfigured to execute the instructions to: receive first image data fromthe first image device and second image data from the second imagedevice; separately store the first image data and the second image data;combine the first image data and the second image data to compositethird image data; and provide the third image data; wherein the thirdimage data is a fused image obtained by fusing, pixel by pixel, thefirst image data and the second image data; wherein the step ofcombining the first image data and the second image data to compositethe third image data comprises: determining a first image property ofthe first image data at a k^(th) pixel position and a second imageproperty of the second image data at the k^(th) pixel position accordingto gray values of the first image data and the second image data at eachpixel position; comparing the first image property with the second imageproperty to determine the gray value at the k^(th) pixel position,wherein k is an integer from 1 to K, wherein K is a number of pixels ofthe first image data and the second image data; and obtaining the fusedimage according to the gray values at all pixel positions; and an imagetransmission apparatus connected to the dual-light image integrationdevice and configured to transmit the third image data composited by thedual-light image integration device to a ground terminal.
 16. The UAVaccording to claim 15, wherein the first image device is an infraredcamera, and the second image device is a high-definition camera.